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In this paper, we review recent developments in transparent 
data embedding and watermarking for audio, image, and video. 
Data-embedding, and watermarking algorithms embed text, binary 
streams, audio, image, or video in a host audio, image, or video 
signal The embedded data are perceptually inaudible or invisible 
to maintain the quality of the source data. The embedded data 
can add features to the host multimedia signal, e.g., multilingual 
soundtracks in a movie, or provide copyright protection. We discuss 
the reliability of data-embedding procedures and their ability to 
deliver new services such as viewing a movie in a given rated 
version from a single multicast stream. We also discuss the issues 
and problems associated with copy and copyright protections and 
assess the viability of current watermarking algorithms as a means 
for protecting copyrighted data. 

Keywords — Copyright protection, data embedding, steganogra- 
phy, watermarking. 



I. Introduction 

The past few years have seen an explosion in the use 
of digital media. Industry is making significant investments 
to deliver digital audio, image, arid video information to 
consumers and customers, A new infrastructure of digital 
audio, image, and video recorders and players, on-line ser- 
vices, and electronic commerce is rapidly being deployed. 
At the same time, major corporations are converting their 
audio, image, and video archives to an electronic form. 
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Digital media offer Several distinct advantages over ana- 
log media: the quality] of digital audio, image, and video 
signals is higher thanj that of their analog counterparts. 
Editing is easy becausje one can access the exact discrete 
locations that should he changed. Copying is simple with 
no loss of fidelity. A copy of a digital media is identical 
to the original. Digital! audio, image, and videos are easily 
transmitted over networked information systems. 

These advantages haye opened up many new possibilities. 
In particular, it is possiible to hide data (information) within 
digital audio, image, and video files, The information is 
hidden in the sense thkt it is perceptually and statistically 
undetectable. With mahy schemes, the hidden information 
can still be recovered if the host signal is compressed, 
edited, or converted from digital to analog format and back. 

As we shall see in Section II, pure analog data-hiding 
techniques had been developed in the past. However, these 
techniques are not as robust as most of the digital data 
hiding techniques that we review in this paper. Furthermore, 
they cannot embed asj much data in a host signal as the 
digital approaches, j 

Digital data embedding has many applications. Foremost 
is passive and active ! copyright protection. Many of the 
inherent advantages of digital signals increase problems 
associated with copyright enforcement For this reason, cre- 
ators and distributors 6f digital data are hesitant to provide 
access to their intellectual property. Digital watermarking 
has been proposed asj a means to identify the owner or 
distributor of digital data. 

Data embedding alsp provides a mechanism for embed- 
ding important control,! descriptive, or reference information 
in a given signal. This information can be used for tracking 
the use of a particular clip, e.g., for pay-per-use appli- 
cations, including billing for commercials and video and 
audio broadcast, as well as Internet electronic commerce of 
digital media. It can b4 used to track audio or visual object 
creation, manipulation!, and modification history within a 
given signal without trie overhead associated with creating 
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he knows that the host signal contains data and is familiar 
with the exact algorithm for embedding the data. Note that 
in some applications, e.g., covert communications, the data 
may also be encrypted prior to insertion in a host signal. 

F. Copyright Protection and Ownership Deadlock 

Data-embedding algorithms may be used to establish 
ownership and distribution of data. In fact, this is the 
application of data embedding or watermarking that has 
received most attention in the literature. Unfortunately, 
most current watermarking schemes are unable to resolve 
rightful ownership of digital data when multiple own- 
ership claims are made, i.e., when a deadlock problem 
arises. The inability of many data-embedding algorithms to 
deal with deadlock, first described by Craver et ai [15], 
is independent of how the watermark is inserted in the 
multimedia data or how robust it is to various types of 
modifications. 

Today, no scheme can unambiguously determine own- 
ership of a given multimedia signal if it does not use an 
original or other copy in the detection process to at least 
construct the watermark to be detected. A pirate can simply 
add his watermark to the watermarked data or counterfeit 
a watermark that correlates well or is detected in the 
contested signal. Current data-embedding schemes used as 
copyright-protection algorithms are unable to establish who 
watermarked the data first. Furthermore, none of the current 
data-embedding schemes has been proven to be immune to 
counterfeiting watermarks that will correlate well with a 
given signal as long as the watermark is not restricted to 
depend partially in a noninvertible manner on the signal. 

If the detection scheme can make use of the original 
to construct. the watermark, then it may be possible to 
establish unambiguous ownership of the data regardless of 
whether the detection scheme subtracts the original from the 
signal under consideration prior to watermark detection or 
not. Specifically, [16] derives a set of sufficient conditions 
that watermarks and watermarking schemes must satisfy to 
provide unambiguous proof of ownership, For example, one 
can use watermarks derived from pseudorandom sequences 
that depend on the signal and the author. Reference [16] 
establishes that this will work for all watermarking proce- 
dures regardless of whether they subtract the original from 
the signal under consideration prior to watermark detection 
or not. Reference [85] independently derived a similar result 
for a restricted class of watermarking techniques that rely 
on subtracting a signal derived from the original from the 
signal under consideration prior to watermark detection. 
The signal-dependent key also helps to thwart the "mix- 
and-match" attack described in [16]. 

An author can construct a watermark that depends on the 
signal and the author and provides unambiguous proof of 
ownership as follows. The author has two random keys xi 
and X2 (i.e., seeds) from which a pseudorandom sequence y 
can be generated using a suitable pseudorandom sequence 
generator [76]. Popular generators include RSA, Rabin, 
Blum/Micali, and BIum/Blum/Shub [25]. With the two 
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proper keys, the watermark may be extracted. Without the 
two keys, the data hidden in the signal are statistically 
undetectable and impossible to recover. Note that classical 
maximal length pseudonoise sequences (i.e., m-sequence) 
generated by linear feedback shift registers are not used 
to generate a watermark. Sequences generated by shift 
registers are cryptographically insecure: one can solve for 
the feedback pattern! (i.e., the keys) given a small number 
of output bits y, \ 

^ The noise-like sequence y may be used to derive the 
actual watermark hidden into the signal or to control the 
operation of the watdrmarking algorithm, e.g., to determine 
the location of pixels that may be modified. The key x x 
is author dependent. The key x 2 is signal dependent. The 
key xi is the secret! key assigned to (or chosen by) the 
author. The key x 2 [is computed from the signal that the 
author wishes to watermark. It is computed from the signal 
using a one-way hash function. For example, the tolerable 
error levels supplied' by masking models (see Section IV) 
are hashed in [85] to a key x 2 . Any one of a number of 
well-known secure one-way hash functions may be used to 
compute x 2t including RSA, MD4 [77], and SHA [60]. For 
example, the Blum/Blum/Shub pseudorandom generator 
uses the one-way function y = g n (x) = x 2 modn 1 where 
n-pq for primes p jand q so that p = q = 3mod4. It can 
be shown that geneijating x or y from partial knowledge- 
of y is computationally infeasible for the Blum/Blum/Shub 
generator. 

The signal-dependtnt key rc 2 makes counterfeiting very 
difficult. The pirate dan only provide key x i to the arbitra- 
tor. Key aj 2 is automatically computed by the watermarking 
algorithm from the original signal. As it is computationally 
infeasible to invert the one-way hash function, the pirate 
is unable to fabricate! a counterfeit original that generates a 
desired or predetermined watermark. 

Deadlock may also be resolved using the dual water- 
marking scheme of | [85]. That scheme employs a pair 
of watermarks. One j watermarking procedure requires the 
original data set for Watermark detection. The second wa- 
termarking procedure; does not require the original data set. 
A data-embedding technique that satisfies the restrictions 
outlined in [16] can bje used to insert the second watermark. 

The above discussion clearly highlights the limitation 
of watermarking as |an unambiguous mean of establish- 
ing ownership. Future clever attacks may show that the 
schemes described ^n [16] or [85] are still vulnerable 
to deadlock. Furthermore, all parties would need to use 
watermarking technidues that have been proven or certified 
to be immune to deadlock to establish ownership of media. 
Note also that contentions of ownership can occur in too 
many different form^. Copyright protection will probably 
not be resolved exclusively by one group or even the 
entire technical community since it involves too many 
legal issues, including the very definition of similarity and 
derived works. Manyjmultidisciplinary efforts are currently 
investigating standards and rules for national and interna- 
tional copyright protection and enforcement in the digital 
age. I 
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Fig. 1, Diagram of a data-embedding algorithm. The information 
is embedded into the signal using the embedding algorithm and 
a key. The dashed lines indicate that the algorithm may directly 
exploit perceptual analysis to embed information. 

IV. Signal Insertion: The Role of Masking 

The first problem that all data-embedding and watermark- 
ing schemes need to address is that of inserting data in the 
digital signal without deteriorating its perceptual quality. 
Of course, we must be able to retrieve the data from the 
edited host signal, i.e., the insertion method must also 
be invertible. Since the data-insertion and data-recovery 
procedures are intimately related, the insertion scheme must 
take into account the requirement of the data-embedding 
application. In many applications, we will need to be able 
to retrieve the data even when the host signal has undergone 
modifications, such as compression, editing, or translation 
between formats, including A/D and D/A conversions. 

Data insertion is possible because the digital medium 
is ultimately consumed by a human. The human hearing 
and visual systems are imperfect detectors. Audio and 
visual signals must have a minimum intensity or contrast 
level before they can be detected by a human. These 
minimum levels depend on the spatial, temporal, and fre- 
quency characteristics of the human auditory and visual 
systems. Further, the human hearing and visual systems are 
characterized by an important phenomenon called masking. 
Masking refers to the fact that a component in a given audio 
or visual signal may become imperceptible in the presence 
of another signal called the masker, Most signal-coding 
techniques (e.g., [41]) exploit the characteristics of the 
human auditory and visual systems directly or indirectly. 
Likewise, all data-embedding techniques exploit the charac- 
teristics of the human auditory and visual systems implicitly 
or explicitly (see Fig. 1). In fact, embedding data would not 
be possible without the limitations of the human visual and 
auditory systems. For example, it is not possible to modify 
a binary stream that represents programs or numbers that 
will be interpreted by a computer. The modification would 
directly and adversely affect the output of the computer. 

A. The Human Auditory System (HAS) 

Audio masking is the effect by which a faint but audible 
sound becomes inaudible in the presence of another louder 
audible sound, i.e., the masker [42]. The masking effect 
depends on the spectral and temporal characteristics of both 
the masked signal and the masker. 

Frequency masking refers to masking between frequency 
components in the audio signal. If two signals that occur 
simultaneously are close together in frequency, the stronger 
masking signal may make the weaker signal inaudible. The 



masking thresholu ^f'a masker depends on the frequency, 
sound pressure level, and tone-like or noise-like character- 
istics of both the masker and the masked signal [61]. It is 
easier for a broad-band nioise to mask a tonal signal than for 
a tonal signal to mask but a broad-band noise. Moreover, 
higher frequency signals are more easily masked. 

The human ear acts as a frequency analyzer and can 
detect sounds with frequencies that vary from 10 to 20000 
Hz. The HAS can be mbdeled by a set of bandpass filters 
with bandwidths that increase with increasing frequency. 
The bands are known as the critical bands. The critical 
bands are defined around a center frequency in which the 
noise bandwidth is increased until there is a just noticeable 
difference in the tone at the center frequency. Thus, if a 
faint tone lies in the critical band of a louder tone, the faint 
tone will not be perceptible. 

Frequency-masking models are readily obtained from the 
current generation of high-quality audio codecs, e.g., the 
masking model defined in the International Standards Or- 
ganization (ISO)-MPEG Audio Psychoacoustic Model 1 for 
Layer I [40]. The Layer I masking method is summarized 
as follows for a 32-kHd sampling rate. The MPEG model 
also supports sampling rates of 44.1 and 48 kHz. 

The frequency mask is computed on localized segments 
(or windows) of the audio signal. The first step consists of 
computing the power spectrum of a short window (512 or 
1024 samples) of the audio signal. Tonal (sinusoidal) and 
nontonal (noisy) components in the spectrum are identified 
because their masking mjodels are different. A tonal compo- 
nent is a local maximum of the spectrum. The auditory sys- 
tem behaves as a bank of bandpass filters, with continuously 
overlapping center frequencies. These "auditory filters" can 
be approximated by rectangular filters with critical band- 
width increasing with frequency. In this model, the audible 
band is therefore divided into 24 nonregular critical bands. 

Next, components below the absolute hearing threshold 
and tonal components iseparated by less than 0.5 Barks 
are removed. The final step consists of computing indi- 
vidual and global masking thresholds. The frequency axis 
is discretized according to hearing sensitivity and express 
frequencies in Barks. Note that hearing sensitivity is higher 
at low frequencies. The resulting masking curves are almost 
linear and depend on a masking index different for tonal and 
nontonal components. They are characterized by different 
lower and upper slopes depending on the distance between 
the masked and the masking component. We use /i to 
denote the set of frequencies present in the test signal. The 
global masking threshold for each frequency f 2 takes into 
account the absolute heiring threshold S a and the masking 
curves P2 of the N t tdnal components and N n nontonal 
components 
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APPENDIX H 

FILING CONTENTS AUTHENTICATION 

Joe Agent creates a document when filing a patent 
application for a client. He wants an additional piece of 
evidence that the patent application was mailed and that 
what was mailed was the patent application. So, he creates 
a PDF file of the application's text and drawings and 
computes the SHAl hash of the PDF file using! some the free 
GPG software ("GNU Privacy 

Guard"), which he trusts. He makes a copy of the Express 
Mail label under which the application is to be mailed 
before having the label initialed and dated by the postal 
worker . 

He then makes a Word 97 document that includes a brief 
block of text for a witness to sign and date. The block of 
text includes the Express Mail label and the hash. The 
document displays the digits of the hash las vertically 
oriented digits in various outline fonts, as I 

with the ACI (See, e.g., Appendix A.) He: then puts the 
photocopy of the non-initialed express 

mail label in his printer's paper supply, and prints the 
word 97 (RTM) document. 

The resulting document bears the image of the non- 
initialed express mail label with the hash of the patent 
application PDF file throughout its background and in a 
block of text of the top. Joe presents the document to Dane 
Attorney, an attorney in the office next door, for her 
dated signature. Now he can present the PDF file and the 
signed and dated paper document as evidence that the patent 
application reproduced in the PDF file existed before the 
express mail label was used to mail a package. This 
provides evidence that the patent application, reproduced 
in the PDF file, was actually what was mailed on the date 
of the express mail label. The proof can be made a bit 



stronger by including the express mail label number in a 
footer on the first page of the patent application, so that 
the PDF file contains the express mail label number shown 
in the document. But that's going way beyond the level of 
evidence currently needed to reconstruct a file lost by the 
PTO. 

This particular embodiment of the inventions can be 
employed any situation where proof of mailing of a 
particular document is desired, without the need for 
digital signatures and the associated hassles of public key 
authentication. The witness to this document doesn't even 

need to know what digital signatures or hash codes are. 
He or she is simply testifying as to the existence of the 
document with that particular code. 
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Multiplicative Group O s Aon ( pse<j<±o^f 0 o^ 
f(x,y) = xy mod 2 A N-k, where x,y A N-k; 

» x where x < 2 A N-k. (y always < 2 A N-k.) 
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Multiplicative Group O Jon ( t>$6Uc!cK2\f Oup>) 
f(x,y) = xy mod 2 A N-k, wi «ere x,y A N-k; -) ' 
■ x where x < 2 A N-k. (y always < 2 A N-k.) 
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Multiplicative Group O Jon ( 05VO&0(X fOUp ) 
f(x,y) = xy mod 2 A N-k, wnere x,y A N-k; ^ 
= x where x < 2 A N-k. (y always < 2 A N-k.) 



N=8 bits 2 N = 256 k = 5 p=251 
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Multiplicative Group O iion (&5tO&0°Sbty ) 
f(x,y) = xy mod 2 A N-k, where x,y K N-k; J 
= x where x < 2 A N-k. (y always < 2 A N-k.) 
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Multiplicative Group O jon ( b^tvdoOi r 014.0 ) 
f(x,y) = xy mod 2 A N-k, wnere x,y *N-k; J ' 
= x where x < 2 A N-k. (y always < 2 A N-k.) 



N = 16 bits 2 N =65536 k= 15 p =65521 prime modulus 

^S£0doyo u p operation with key y1: y j =6595 z } =f(x r y ,) 
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Multiplicative Group O Jopv ( p<a)do(\ fo Op) 
f(x,y) = xymod2 A N-k,v.„erex.y A N-k; O • 
= x where x < 2 A N-k. (y always < 2 A N-k.) 

N = 16 bits 2 N = 65536 k= 15 p =65521 prime modulus 
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Part 2: Commercial Cryptographic 



. * J?-? 51 ? jested that increasing the IDEA subbi 

e fu i^ 0CkS fr ° m de8ien of 16 bits wouI <* ^creaae the « 
of the IDEA algorithm to a factor of 2". Lai answered that the i 
of the algonthm was based on the fact that 2" + 1 is a prime » 
2 +1 is not. Lai suggests that the stronger properties of the all 
would be compromised. The point is that small changes in strucfe 
have adverse ripple effects on the cryptographic structure ft" 
k, ^ \'. » I \ becomese ^ us Cementation errors. We look at other implemen 
N*V So liM\t t c5 errors m Chapter 13. v^men 
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TABLE 1-1: "Pseudogroup" Operation: p=11 (prime), m=3 bits 
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TABLE I-2: Key values amo VyA«> <x>soa*4<:c( ^j-Vk-tWv 

Each row contains in p«v values producing a given output, except for holes (black squares). 

Each column contains key values produc ing outputs for a given input, except for holes. 
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A "hole" is an output value that will not occur for any in the set {1 ,2,...2 A m} of possible key 
values, given a particular input value in that set. 
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TABLE IM:fAQjX££r) Multiplicative Group Operation x*y mod p, p=17 (prime), m=4 bits 
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TABLE 11-2: SVo^c-fc Operation: p=19 (prime), m=4 bits 
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Each row conta.ns Rvalues producing a given output except for holeT(black squares) 
Each column contains key values producing outputs for a given input, except for holes 
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holes. m 



function h = holes (y, p,M, short) ; 

% h = holes (y,p,N, short); 

% CONFIDENTIAL AND PROPRIETARY 

% Edwin A. Suominen 

% Finds "holes" - skipped values of set {0,1} A N ih result 
% of x*y mod p. 

% Number of values in set 3: {0,1} A N 
% M = 2 A N; 

% For vector inputs. . . 
for k=l : length (y) 

s = 1:M; % Working array of values in set S 

% Zero out values in set that occur ("non-holes") 
for i = 1:M 

j = product (i,y(k) ,p) ; % xy mod p 

% Zero out if not a hole 

if j<=M, s(j) - 0/ end 
end 

% Sort decending to get holes first 
z » -sort C-s) ; 

if nargin>3, 
% There can be no more than p-2 A N-l holes. 
% Limit size of result vector (s) accordincrlv 
z = z (l:p-M-l) ; * * 

% Add result vector to array (if vector y) 
h(:,k) = z f / y 
else 

h(:,k) = s'; 
end 



end 



end 



holeplot . 



function [yl,y2] = holeplot (p,M) ; 

% y = holeplot (p,M) 

% Modulus is 2 A N+k, where k is odd 
% p = M + k; 

% Create 1010 .. .matrix of holes 
for i = l:M/2, 

% Get holes for each column 

s « holes (M+l-i,p,M) / 

% Convert to 1010... format 

s = s>0; 

% Convert to text string 
col = M+l-i 
if col<10 f 

rl = [' ' num2str (col, 2) ] / 
else 

rl = nura2str (col, 2) ; 
end 

yl(i,:> = [ rl '- T num2str <s, 1) ] ; 
end 

for i - l:M/2, 

% Get holes for each column 
s = holes (M+l- (i+M/2) ,p,M) ; 
% Convert to 1010... format 
S = s>0; 

% Convert to text string 
col - M/2+l-i 
if coKlO, 

rl * [■ T num2str (col, 2) ] ; 
else 

rl = num2str (col, 2) ; 
end 

y2(i,:) = [ rl num2atr (s, 1) ] ; 

end 
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